Spatial-Temporal Semantic Grouping of Instructional Video Content

This paper presents a new approach for content analysis and semantic summarization of instructional videos of blackboard presentations. We first use low-level image processing techniques to segment frames into board content regions, regions occluded by instructors, and irrelevant areas, then measure the number of chalk pixels in the content areas of each frame. Using the number of chalk pixels as heuristic measurement of video content, we derive a content figure which describes the actual rather than apparent fluctuation of video content. By searching for local maxima in the content figure, and by detecting camera motions and tracking movements of instructors, we can then define and retrieve key frames. Since some video content may not be contained in any one of the key frames due to occlusion by instructors or camera motion, we use an image registration method to make “board content images” that are free of occlusions and not bound by frame boundaries. Extracted key frames and board content images are combined together to summarize and index the video. We further introduce the concept of “semantic teaching unit”, which is defined as a more natural semantic temporalspatial unit of teaching content. We propose a model to detect semantic teaching units, based on the recognition of actions of instructors, and on the measurement of temporal duration and spatial location of board content. We demonstrate experiments on instructional videos which are taken in non-instrumented classrooms, and show examples of the construction of board content images and the detection of semantic teaching units within them.